Vector Symbolic Architecture Research Articles

A Two-Layer Self-Organizing Map with Vector Symbolic Architecture for Spatiotemporal Sequence Learning and Prediction.

We propose a new nature- and neuro-science-inspired algorithm for spatiotemporal learning and prediction based on sequential recall and vector symbolic architecture. A key novelty is the learning of spatial and temporal patterns as decoupled concepts where the temporal pattern sequences are constructed using the learned spatial patterns as an alphabet of elements. The decoupling, motivated by cognitive neuroscience research, provides the flexibility for fast and adaptive learning with dynamic changes to data and concept drift and as such is better suited for real-time learning and prediction. The algorithm further addresses several key computational requirements for predicting the next occurrences based on real-life spatiotemporal data, which have been found to be challenging with current state-of-the-art algorithms. Firstly, spatial and temporal patterns are detected using unsupervised learning from unlabeled data streams in changing environments; secondly, vector symbolic architecture (VSA) is used to manage variable-length sequences; and thirdly, hyper dimensional (HD) computing-based associative memory is used to facilitate the continuous prediction of the next occurrences in sequential patterns. The algorithm has been empirically evaluated using two benchmark and three time-series datasets to demonstrate its advantages compared to the state-of-the-art in spatiotemporal unsupervised sequence learning where the proposed ST-SOM algorithm is able to achieve 45% error reduction compared to HTM algorithm.

Conjunctive block coding for hyperdimensional graph representation

Knowledge Graphs (KGs) have become a pivotal knowledge representation tool in machine learning, not only providing access to existing knowledge but also enabling the discovery of new knowledge through advanced applications. Among the scalable reasoning methods used for such applications, distributed graph embedding approaches, particularly GNNs, have become popular for large-scale graph-related tasks. However, many of these methods have limitations in their interpretability and fail to take into account structural similarity in their representation. Hyperdimensional Computing (HDC), also known as Vector Symbolic Architecture (VSA), addresses this issue by using well-defined cognitive operations on distributed representations of symbolic concepts. This work proposes and evaluates a new vector symbolic graph representation, CLOG, that preserves approximate structural similarity beyond edge correspondence and fundamentally differs from previous methods. The model's effectiveness in graph representation is evaluated through theoretical analysis, graph reconstruction experiments, and link prediction task, highlighting its efficiency and accuracy. This approach significantly advances the field by enhancing the capabilities of HDC in graph representation, representing a notable improvement over existing methods.

Testing and Enhancing Adversarial Robustness of Hyperdimensional Computing

Brain-inspired hyperdimensional computing (HDC), also known as Vector Symbolic Architecture (VSA), is an emerging “non-von Neumann” computing scheme that imitates human brain functions to process information or perform learning tasks using abstract and high-dimensional patterns. Compared with deep neural networks (DNNs), HDC shows advantages such as compact model size, energy efficiency, and few-shot learning. Despite of those advantages, one under-investigated area of HDC is the adversarial robustness; existing works have shown that HDC is vulnerable to adversarial attacks where attackers can add minor perturbations onto the original inputs to “fool” HDC models, producing wrong predictions. In this paper, we systematically study the adversarial robustness of HDC by developing a systematic approach to test and enhance the robustness of HDC against adversarial attacks with two main components: (1), which is a highly-automated testing tool that can generate high-quality adversarial data for a given HDC model; and (2), which utilizes the adversarial data generated by to enhance the adversarial robustness of HDC models. The core idea of is built on top of fuzz testing method. We customize the fuzzing approach by proposing a similarity-based coverage metric to guide to continuously mutate original inputs to generate new inputs that can trigger incorrect behaviors of HDC model. Thanks to the use of differential testing, does not require knowing the labels of the samples beforehand. For enhancing the adversarial robustness, we design, implement, and evaluate to defend HDC models against adversarial data. The core idea of is an adversarial detector which can be trained by -generated adversarial samples. During inference, once an adversarial sample is detected, will override the prediction result with an “invalid” signal. We evaluate the proposed methods on 4 datasets and 5 adversarial attack scenarios with 6 adversarial generation strategies and 2 defense mechanisms, and compare the performance correspondingly. is able to differentiate between benign and adversarial inputs with over 90% accuracy, which is up to 55% higher than adversarial training-based baselines. To the best of our knowledge, this paper presents the first comprehensive effort in systematically testing and enhancing the robustness against adversarial data of this emerging brain-inspired computational model.

Associative memory of structured knowledge

A long standing challenge in biological and artificial intelligence is to understand how new knowledge can be constructed from known building blocks in a way that is amenable for computation by neuronal circuits. Here we focus on the task of storage and recall of structured knowledge in long-term memory. Specifically, we ask how recurrent neuronal networks can store and retrieve multiple knowledge structures. We model each structure as a set of binary relations between events and attributes (attributes may represent e.g., temporal order, spatial location, role in semantic structure), and map each structure to a distributed neuronal activity pattern using a vector symbolic architecture scheme.We then use associative memory plasticity rules to store the binarized patterns as fixed points in a recurrent network. By a combination of signal-to-noise analysis and numerical simulations, we demonstrate that our model allows for efficient storage of these knowledge structures, such that the memorized structures as well as their individual building blocks (e.g., events and attributes) can be subsequently retrieved from partial retrieving cues. We show that long-term memory of structured knowledge relies on a new principle of computation beyond the memory basins. Finally, we show that our model can be extended to store sequences of memories as single attractors.

Trustable service discovery for highly dynamic decentralized workflows

The quantity and capabilities of smart devices and sensors deployed as part of the Internet of Things (IoT) and accessible via remote microservices is set to rise dramatically as the provision of interactive data streaming increases. This introduces opportunities to rapidly construct new applications by interconnecting these microservices in different workflow configurations. The challenge is to discover the required microservices, including those from trusted partners and the wider community, whilst being able to operate robustly under diverse networking conditions. This paper outlines a workflow approach that provides decentralized discovery and orchestration of verifiably trustable services in support of multi-party operations. The approach is based on adoption of patterns from self-sovereign identity research, notably Verifiable Credentials, to share information amongst peers based on attestations of service descriptions and prior service usage in a privacy preserving and secure manner. This provides a dynamic, trust-based framework for ratifying and evaluating the qualities of different services. Collating these new service descriptions and integrating with existing decentralized workflow research based on vector symbolic architecture (VSA) provides an enhanced semantic search space for efficient and trusted service discovery that is necessary to support a diverse range of emerging edge-computing environments. An architecture for a dynamic decentralized service discovery system, is designed, and described through application to a scenario which uses trusted peers’ reported experiences of an anomaly detection service to determine service selection.

Vector Semiotic Model for Visual Question Answering

In this paper, we propose a Vector Semiotic Model as a possible solution to the symbol grounding problem in the context of Visual Question Answering. The Vector Semiotic Model combines the advantages of a Semiotic Approach implemented in the Sign-Based World Model and Vector Symbolic Architectures. The Sign-Based World Model represents information about a scene depicted on an input image in a structured way and grounds abstract objects in an agent’s sensory input. We use the Vector Symbolic Architecture to represent the elements of the Sign-Based World Model on a computational level. Properties of a high-dimensional space and operations defined for high-dimensional vectors allow encoding the whole scene into a high-dimensional vector with the preservation of the structure. That leads to the ability to apply explainable reasoning to answer an input question. We conducted experiments are on a CLEVR dataset and show results comparable to the state of the art. The proposed combination of approaches, first, leads to the possible solution of the symbol-grounding problem and, second, allows expanding current results to other intelligent tasks (collaborative robotics, embodied intellectual assistance, etc.).

Reasoning and learning with context logic

Context logic (CL), a logical language similar in style to description logics but with a more cognitive motivation as a logical language of cognition, was developed since 2007 to provide a new approach to the symbol grounding problem, a key problem for reliable intelligent environments and other intelligent sensory systems. CL is a three-layered integrated hierarchy of languages: a relational base layer with the expressiveness of propositional logic (CLA), a quantifier-free decidable language (CL0), and an expressive language with full quantification (CL1). As was shown in 2018, the core CLA reasoning can be implemented on a variant of Kanerva’s Vector Symbolic Architecture, the activation bit vector machine (ABVM), shedding new light on the fundamental cognitive faculties of symbol grounding and imagery, but the system raised two questions: first, the core reasoning algorithm was a classical EXPTIME reasoner; second, fundamental aspects for a learning algorithm were sketched but not presented with a full algorithm. This paper addresses those two questions. We present a probabilistic linear time algorithm for reasoning over conjunctive normal form (CNF) CLA formulae together with a dual probabilistic linear time algorithm for learning CLA statements by collecting experienced snapshots in a disjunctive normal form (DNF).

Efficient orchestration of Node-RED IoT workflows using a Vector Symbolic Architecture

Numerous workflow systems span multiple scientific domains and environments, and for the Internet of Things (IoT), Node-RED offers an attractive Web based user interface to execute IoT service-based workflows. However, like most workflow systems, it coordinates the workflow centrally, and cannot run within more transient environments where nodes are mobile. To address this gap, we show how Node-RED workflows can be migrated into a decentralized execution environment for operation on mobile ad-hoc networks, and we demonstrate this by converting a Node-RED based traffic congestion detection workflow to operate in a decentralized environment. The approach uses a Vector Symbolic Architecture (VSA) to dynamically convert Node-Red applications into a compact semantic vector representation that encodes the service interfaces and the workflow in which they are embedded. By extending existing services interfaces, with a simple cognitive layer that can interpret and exchange the vectors, we show how the required services can be dynamically discovered and interconnected into the required workflow in a completely decentralized manner. The resulting system provides a convenient environment where the Node-RED front-end graphical composition tool can be used to orchestrate decentralized workflows. In this paper, we further extend this work by introducing a new dynamic VSA vector compression scheme that compresses vectors for on-the-wire communication, thereby reducing communication bandwidth while maintaining the semantic information content. This algorithm utilizes the holographic properties of the symbolic vectors to perform compression taking into consideration the number of combined vectors along with similarity bounds that determine conflict with other encoded vectors used in the same context. The resulting savings make this approach extremely efficient for discovery in service based decentralized workflows.

Connecting Twenty-First Century Connectionism and Wittgenstein

By pointing to deep philosophical confusions endemic to cognitive science, Wittgenstein might seem an enemy of computational approaches. We agree (with Mills 1993) that while Wittgenstein would reject the classicist’s symbols and rules approach, his observations align well with connectionist or neural network approaches. While many connectionisms that dominated the later twentieth century could fall prey to criticisms of biological, pedagogical, and linguistic implausibility, current connectionist approaches can resolve those problems in a Wittgenstein-friendly manner. We (a) present the basics of a Vector Symbolic Architecture formalism, inspired by Smolensky (1990), and indicate how high-dimensional vectors can operate in a context-sensitive and object-independent manner in biologically plausible time scales, reflecting Wittgenstein’s notions of language-games and family resemblance; we (b) show how “soft” symbols for such a formalism can be formed with plausible learning cycles using Sparse Distributed Memory, resolving disputes surrounding Wittgenstein’s private language argument; and (c) show how connectionist networks can extrapolate meaningful patterns to solve problems, providing “ways to go on” without explicit rules, which indicates linguistic plausibility. Connectionism thus provides a systematicity and productivity that is more than a mere implementation of a classical approach, and provides Wittgenstein-friendly and Wittgenstein-illuminating models of mind and language for cognitive science.

Semantic Similarity Estimation Using Vector Symbolic Architectures

For many natural language processing applications, estimating similarity and relatedness between words are key tasks that serve as the basis for classification and generalization. Currently, vector semantic models (VSM) have become a fundamental language modeling tool. VSMs represent words as points in a high-dimensional space and follow the distributional hypothesis of meaning, which assumes that semantic similarity is related to the context. In this paper, we propose a model whose representations are based on the semantic features associated with a concept within the ConceptNet knowledge graph. The proposed model is based on a vector symbolic architecture framework, which defines a set of arithmetic operations to encode the semantic features within a single high-dimensional vector. In addition to word distribution, these vector representations consider several types of information. Moreover, owing to the properties of high-dimensional spaces, they have the additional advantage of being interpretable. We analyze the model's performance on the SimLex-999 dataset, a dataset where commonly used distributional models (e.g., word2vec or GloVe) perform poorly. Our results are similar to those of other hybrid models, and they surpass several state-of-the-art distributional and knowledge-based models.

Constructing distributed time-critical applications using cognitive enabled services

Abstract Time-critical analytics applications are increasingly making use of distributed service interfaces (e.g., micro-services) that support the rapid construction of new applications by dynamically linking the services into different workflow configurations. Traditional service-based applications, in fixed networks, are typically constructed and managed centrally and assume stable service endpoints and adequate network connectivity. Constructing and maintaining such applications in dynamic heterogeneous wireless networked environments, where limited bandwidth and transient connectivity are commonplace, presents significant challenges and makes centralized application construction and management impossible. In this paper we present an architecture which is capable of providing an adaptable and resilient method for on-demand decentralized construction and management of complex time-critical applications in such environments. The approach uses a Vector Symbolic Architecture (VSA) to compactly represent an application as a single semantic vector that encodes the service interfaces, workflow, and the time-critical constraints required. By extending existing services interfaces, with a simple cognitive layer that can interpret and exchange the vectors, we show how the required services can be dynamically discovered and interconnected in a completely decentralized manner. We demonstrate the viability of this approach by using a VSA to encode various time-critical data analytics workflows. We show that these vectors can be used to dynamically construct and run applications using services that are distributed across an emulated Mobile Ad-Hoc Wireless Network (MANET). Scalability is demonstrated via an empirical evaluation.

Vector space architecture for emergent interoperability of systems by learning from demonstration

The rapid integration of physical systems with cyberspace infrastructure, the so-called Internet of Things, is likely to have a significant effect on how people interact with the physical environment and design information and communication systems. Internet-connected systems are expected to vastly outnumber people on the planet in the near future, leading to grand challenges in software engineering and automation in application domains involving complex and evolving systems. Several decades of artificial intelligence research suggests that conventional approaches to making such systems automatically interoperable using handcrafted “semantic” descriptions of services and information are difficult to apply. In this paper we outline a bioinspired learning approach to creating interoperable systems, which does not require handcrafted semantic descriptions and rules. Instead, the idea is that a functioning system (of systems) can emerge from an initial pseudorandom state through learning from examples, provided that each component conforms to a set of information coding rules. We combine a binary vector symbolic architecture (VSA) with an associative memory known as sparse distributed memory (SDM) to model context-dependent prediction by learning from examples. We present simulation results demonstrating that the proposed architecture can enable system interoperability by learning, for example by human demonstration.

On Bidirectional Transitions between Localist and Distributed Representations: The Case of Common Substrings Search Using Vector Symbolic Architecture

The contribution of this article is twofold. First, it presents an encoding approach for seamless bidirectional transitions between localist and distributed representation domains. Second, the approach is demonstrated on the example of using Vector Symbolic Architecture for solving a problem of finding common substrings. The proposed algorithm uses elementary operations on long binary vectors. For the case of two patterns with respective lengths L1 and L2 it requires Θ(L1 + L2 – 1) operations on binary vectors, which is equal to the suffix trees approach – the fastest algorithm for this problem. The simulation results show that in order to be robustly detected by the proposed approach the length of a common substring should be more than 4% of the longest pattern.

Realistic neurons can compute the operations needed by quantum probability theory and other vector symbolic architectures

Quantum probability (QP) theory can be seen as a type of vector symbolic architecture (VSA): mental states are vectors storing structured information and manipulated using algebraic operations. Furthermore, the operations needed by QP match those in other VSAs. This allows existing biologically realistic neural models to be adapted to provide a mechanistic explanation of the cognitive phenomena described in the target article by Pothos & Busemeyer (P&B).

Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors

The 1990s saw the emergence of cognitive models that depend on very high dimensionality and randomness. They include Holographic Reduced Representations, Spatter Code, Semantic Vectors, Latent Semantic Analysis, Context-Dependent Thinning, and Vector-Symbolic Architecture. They represent things in high-dimensional vectors that are manipulated by operations that produce new high-dimensional vectors in the style of traditional computing, in what is called here hyperdimensional computing on account of the very high dimensionality. The paper presents the main ideas behind these models, written as a tutorial essay in hopes of making the ideas accessible and even provocative. A sketch of how we have arrived at these models, with references and pointers to further reading, is given at the end. The thesis of the paper is that hyperdimensional representation has much to offer to students of cognitive science, theoretical neuroscience, computer science and engineering, and mathematics.