Part-whole Hierarchies Research Articles

Agglomerator++: Interpretable part-whole hierarchies and latent space representations in neural networks

Deep neural networks achieve outstanding results in a large variety of tasks, often outperforming human experts. However, a known limitation of current neural architectures is the poor accessibility in understanding and interpreting the network’s response to a given input. This is directly related to the huge number of variables and the associated non-linearities of neural models, which are often used as black boxes. This lack of transparency, particularly in crucial areas like autonomous driving, security, and healthcare, can trigger skepticism and limit trust, despite the networks’ high performance. In this work, we want to advance the interpretability in neural networks. We present Agglomerator++, a framework capable of providing a representation of part-whole hierarchies from visual cues and organizing the input distribution to match the conceptual-semantic hierarchical structure between classes. We evaluate our method on common datasets, such as SmallNORB, MNIST, FashionMNIST, CIFAR-10, and CIFAR-100, showing that our solution delivers a more interpretable model compared to other state-of-the-art approaches. Our code is available at https://mmlab-cv.github.io/Agglomeratorplusplus/.

Object-centric Learning with Capsule Networks: A Survey

Capsule networks emerged as a promising alternative to convolutional neural networks for learning object-centric representations. The idea is to explicitly model part-whole hierarchies by using groups of neurons called capsules to encode visual entities, then learn the relationships between these entities dynamically from data. However, a major hurdle for capsule network research has been the lack of a reliable point of reference for understanding their foundational ideas and motivations. This survey provides a comprehensive and critical overview of capsule networks, which aims to serve as a main point of reference going forward. To that end, we introduce the fundamental concepts and motivations behind capsule networks, such as equivariant inference . We then cover various technical advances in capsule routing algorithms as well as alternative geometric and generative formulations. We provide a detailed explanation of how capsule networks relate to the attention mechanism in Transformers and uncover non-trivial conceptual similarities between them in the context of object-centric representation learning. We also review the extensive applications of capsule networks in computer vision, video and motion, graph representation learning, natural language processing, medical imaging, and many others. To conclude, we provide an in-depth discussion highlighting promising directions for future work.

A sensory-motor theory of the neocortex.

Recent neurophysiological and neuroanatomical studies suggest a close interaction between sensory and motor processes across the neocortex. Here, I propose that the neocortex implements active predictive coding (APC): each cortical area estimates both latent sensory states and actions (including potentially abstract actions internal to the cortex), and the cortex as a whole predicts the consequences of actions at multiple hierarchical levels. Feedback from higher areas modulates the dynamics of state and action networks in lower areas. I show how the same APC architecture can explain (1) how we recognize an object and its parts using eye movements, (2) why perception seems stable despite eye movements, (3) how we learn compositional representations, for example, part-whole hierarchies, (4) how complex actions can be planned using simpler actions, and (5) how we form episodic memories of sensory-motor experiences and learn abstract concepts such as a family tree. I postulate a mapping of the APC model to the laminar architecture of the cortex and suggest possible roles for cortico-cortical and cortico-subcortical pathways.

Hierarchical Graph Capsule Networks for Molecular Function Classification With Disentangled Representations.

In biochemistry, graph structures have been widely used for modeling compounds, proteins, functional interactions, etc. A common task that divides these graphs into different categories, known as graph classification, highly relies on the quality of the representations of graphs. With the advance in graph neural networks, message-passing-based methods are adopted to iteratively aggregate neighborhood information for better graph representations. These methods, though powerful, still suffer from some shortcomings. The first challenge is that pooling-based methods in graph neural networks may sometimes ignore the part-whole hierarchies naturally existing in graph structures. These part-whole relationships are usually valuable for many molecular function prediction tasks. The second challenge is that most existing methods do not take the heterogeneity embedded in graph representations into consideration. Disentangling the heterogeneity will increase the performance and interpretability of models. This paper proposes a graph capsule network for graph classification tasks with disentangled feature representations learned automatically by well-designed algorithms. This method is capable of, on the one hand, decomposing heterogeneous representations to more fine-grained elements, whilst on the other hand, capturing part-whole relationships using capsules. Extensive experiments performed on several public-available biochemistry datasets demonstrated the effectiveness of the proposed method, compared with nine state-of-the-art graph learning methods.

Active Predictive Coding: A Unifying Neural Model for Active Perception, Compositional Learning, and Hierarchical Planning.

There is growing interest in predictive coding as a model of how the brain learns through predictions and prediction errors. Predictive coding models have traditionally focused on sensory coding and perception. Here we introduce active predictive coding (APC) as a unifying model for perception, action, and cognition. The APC model addresses important open problems in cognitive science and AI, including (1) how we learn compositional representations (e.g., part-whole hierarchies for equivariant vision) and (2) how we solve large-scale planning problems, which are hard for traditional reinforcement learning, by composing complex state dynamics and abstract actions from simpler dynamics and primitive actions. By using hypernetworks, self-supervised learning, and reinforcement learning, APC learns hierarchical world models by combining task-invariant state transition networks and task-dependent policy networks at multiple abstraction levels. We illustrate the applicability of the APC model to active visual perception and hierarchical planning. Our results represent, to our knowledge, the first proof-of-concept demonstration of a unified approach to addressing the part-whole learning problem in vision, the nested reference frames learning problem in cognition, and the integrated state-action hierarchy learning problem in reinforcement learning.

Recursive neural programs: A differentiable framework for learning compositional part-whole hierarchies and image grammars.

Human vision, thought, and planning involve parsing and representing objects and scenes using structured representations based on part-whole hierarchies. Computer vision and machine learning researchers have recently sought to emulate this capability using neural networks, but a generative model formulation has been lacking. Generative models that leverage compositionality, recursion, and part-whole hierarchies are thought to underlie human concept learning and the ability to construct and represent flexible mental concepts. We introduce Recursive Neural Programs (RNPs), a neural generative model that addresses the part-whole hierarchy learning problem by modeling images as hierarchical trees of probabilistic sensory-motor programs. These programs recursively reuse learned sensory-motor primitives to model an image within different spatial reference frames, enabling hierarchical composition of objects from parts and implementing a grammar for images. We show that RNPs can learn part-whole hierarchies for a variety of image datasets, allowing rich compositionality and intuitive parts-based explanations of objects. Our model also suggests a cognitive framework for understanding how human brains can potentially learn and represent concepts in terms of recursively defined primitives and their relations with each other.

‘Organism’ Versus ‘Biological Individual’: The Missing Demarcation

The demarcation of organisms from other biological individuals has received relatively little attention. In this paper, I extricate and systematize the different ways in which the organism–biological individual relationship has been construed: (1) coalescence of the two concepts, (2) biological individual eliminativism, (3) organism eliminativism, (4) organism as a ‘paradigmatic’ biological individual, (5) organism as a limit state towards which biological individuals tend in evolution and development, (6) organism as instantiating the whole in a part-whole hierarchy of biological individuals, (7) organism as equivalent to physiological individual, and (8) organism as a special kind of physiological individual. I show that, in most of these stances, the organism concept is too imprecise to be demarcated from other biological individuals, which fosters some form of eliminativism. I also argue that the comparisons between organisms and biological individuals are performed in two different modes: ‘horizontally’ (i.e., between individuals not related hierarchically) or ‘vertically’ (i.e., between individuals belonging to different levels within the same hierarchy). Finally, I explain the challenges that each of these comparison modes face and suggest that the ‘vertical’ mode adumbrates a potential way forward.

How to Represent Part-Whole Hierarchies in a Neural Network

This article does not describe a working system. Instead, it presents a single idea about representation that allows advances made by several different groups to be combined into an imaginary system called GLOM.1 The advances include transformers, neural fields, contrastive representation learning, distillation, and capsules. GLOM answers the question: How can a neural network with a fixed architecture parse an image into a part-whole hierarchy that has a different structure for each image? The idea is simply to use islands of identical vectors to represent the nodes in the parse tree. If GLOM can be made to work, it should significantly improve the interpretability of the representations produced by transformer-like systems when applied to vision or language.

Engaging Part-Whole Hierarchies and Contrast Cues for Salient Object Detection

Real-world scenes always exhibit objects with clutter backgrounds, posing great challenges for deep salient object detection models. In this paper, we propose salient object detection by engaging two saliency cues, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the part-whole hierarchies and contrast cues, resulting in a PWHCNet. Specifically, two branches, which consists of a Dynamic Grouping Capsules (DGC) branch and a DenseHRNet branch, are put in place to learn the part-whole hierarchies and contrast cues, respectively. Moreover, to help highlight the whole salient object in complex scenes, a Background Suppression (BS) module is proposed to guide the shallow features of DenseHRNet with the aid of the part-whole relational cues captured by DGC. Subsequently, these two saliency cues are integrated via a Self-Channel and Mutual-Spatial (SCMS) attention mechanism. Experimental results on five benchmarks demonstrate that the proposed PWHCNet achieves state-of-the-art performance while obtaining the whole salient objects with fine details.

Domain object hierarchies inducing multi-level models

Conceptual modeling of domain object hierarchies, such as product hierarchies or organization hierarchies, is difficult due to the intricate nature of nonphysical domain objects organized in such hierarchies. Modeling domain object hierarchies as part-whole hierarchies covers their hierarchical structure, yet to capture their meaning, part-whole hierarchies have to be combined with specialization and multi-level instantiation. To this end we introduce the deep domain object (DDO) multi-level modeling pattern and approach. With the DDO approach, subclasses and metaclasses are induced by and integrated with the part-whole hierarchy. The approach is aligned with the multi-level theory (MLT) and formalized by a metamodel and a set of deductive rules implemented in F-Logic. The proof-of-concept prototype is used for automated application of the pattern and for querying induced multi-level models.

The Potential of the SP System in Machine Learning and Data Analysis for Image Processing

This paper aims to describe how pattern recognition and scene analysis may with advantage be viewed from the perspective of the SP system (meaning the SP theory of intelligence and its realisation in the SP computer model (SPCM), both described in an appendix), and the strengths and potential of the system in those areas. In keeping with evidence for the importance of information compression (IC) in human learning, perception, and cognition, IC is central in the structure and workings of the SPCM. Most of that IC is achieved via the powerful concept of SP-multiple-alignment, which is largely responsible for the AI-related versatility of the system. With examples from the SPCM, the paper describes: how syntactic parsing and pattern recognition may be achieved, with corresponding potential for visual parsing and scene analysis; how those processes are robust in the face of errors in input data; how in keeping with what people do, the SP system can “see” things in its data that are not objectively present; the system can recognise things at multiple levels of abstraction and via part-whole hierarchies, and via an integration of the two; the system also has potential for the creation of a 3D construct from pictures of a 3D object from different viewpoints, and for the recognition of 3D entities.

Incorporating part-whole hierarchies into fully convolutional network for scene parsing

In this paper, a new approach to scene parsing is proposed which integrates part-whole hierarchies relationship in the last feature map to assign a semantic class label to each pixel. Recently, deep learning-based approaches have had a great impact on scene parsing. However, these methods could not preserve the spatial information about the high-level (or mid-level) features. Hence, Hinton, one of the fathers of deep learning, introduced the capsule concept to encode pose information such as orientation. All of the capsules which have a similar pose matrix value are grouped to form a parent capsule. However, their work has two challenges: 1) the extensive time required to perform dynamic routing agreement to obtain the routing coefficient and 2) the variation of the appearance and the spatial hierarchies between part capsules and their corresponding parent are not encoded. In this study, to consider these challenges, the general Hough transform (GHT) and tensor normal distribution are utilized to propose a novel capsule concept. In this case, each capsule has k offset vectors for each semantic class. The offset vectors are oriented from the capsule to the k other capsules which have an effective role in assigning that capsule to a specific semantic class. The problem formulation is proposed such that evaluating the approach on large datasets is feasable. Also, a new score function is designed to accumulate the vote’s strengths for capsule class estimation. To do so, we use tensor normal distribution in which the covariance matrix is defined as the Kronecker product of the capsule feature covariance and the between-capsule covariance. The proposed approach, for the first time, encodes the relations between part capsules to vote to a whole capsule through the between-capsule covariance matrix. To evaluate our proposed approach, it is applied to SiftFlow, NYUD-v2 and PASCAL VOC 2012 datasets. The results show that our approach achieves superior performance.

SNOMED reaching its adolescence: Ontologists’ and logicians’ health check

After a critical review of the present architecture of SNOMED CT, addressing both logical and ontological issues, we present a roadmap toward an overall improvement and recommend the following actions: SNOMED CT's ontology, dictionary, and information model components should be kept separate. SNOMED CT's upper level should be re-arranged according to a standard upper level ontology. SNOMED CT concepts should be assigned to the four disjoint groups: classes, instances, relations, and meta-classes. SNOMED CT's binary relations should be reduced to a set of canonical ones, following existing recommendations. Taxonomies should be cleansed and split into disjoint partitions. The number of full definitions should be increased. Finally, new approaches are proposed for modeling part-whole hierarchies, as well as the integration of qualifier relations into a unified framework. All proposed modifications can be expressed by the computationally tractable description logic E L + + .

Towards an intelligent database system founded on the SP theory of computing and cognition

The SP theory of computing and cognition, described in previous publications, is an attractive model for intelligent databases because it provides a simple but versatile format for different kinds of knowledge, it has capabilities in artificial intelligence, it can function effectively in the face of errors in its input data, and it can function like established database models when that is required. This paper first describes the SP theory in outline and the computer models in which it is expressed. The main sections of the paper describe, with examples from the SP62 computer model, how the SP framework can emulate other abstract models used in database applications: the relational model (including retrieval of information in the manner of query-by-example, creating a join between two or more tables, and aggregation), object-oriented models (including class-inclusion hierarchies, part-whole hierarchies and their integration, inheritance of attributes, cross-classification and multiple inheritance), and hierarchical and network models (including discrimination networks). Comparisons are made between the SP model and those other models. The artificial intelligence capabilities of the SP model are briefly reviewed: representation and integration of diverse kinds of knowledge in one versatile format; fuzzy pattern recognition and recognition at multiple levels of abstraction; best-match and semantic forms of information retrieval; various kinds of exact reasoning and probabilistic reasoning; analysis and production of natural language; planning; problem solving; and unsupervised learning. Also considered are ways in which current prototypes may be translated into an ‘industrial strength’ working system.

Complexity science in collaborative design

Bringing designers together as successful design teams is investigated in the context of the science of complex systems. The relational structure inherent in design defines multilevel multidimensional networks, which can represent part-whole hierarchies in design. Design is presented as an iterative top-down and bottom-up process that induces the emergence of desirable properties. This involves the construction and instantiation of multilevel representations. The implications of this are discussed for computer supported collaborative design, and how interaction structures can be designed to support design collaboration. The paper is aimed at a general reader and is self-contained.

Medical knowledge reengineering—converting major portions of the UMLS into a terminological knowledge base

We describe a semi-automatic knowledge engineering approach for converting the human anatomy and pathology portion of the UMLS metathesaurus into a terminological knowledge base. Particular attention is paid to the proper representation of part-whole hierarchies, which complement taxonomic ones as a major hierarchy-forming principle for anatomical knowledge. Our approach consists of four steps. First, concept definitions are automatically generated from the metathesaurus, with loom as the target language. Second, integrity checking of the emerging taxonomic and partonomic hierarchies is automatically carried out by the terminological classifier. Third, terminological cycles and inconsistencies are manually eliminated and, in the last step, the knowledge base built this way is incrementally refined by a medical expert. Our experiments were run on a terminological knowledge base which is composed of 164 000 concepts and 76 000 relations. Empirical evidence for the lack of logical consistency, adequacy and improper granularity of the UMLS knowledge source is given, and finally, assessments of what kind of efforts are needed to render the formal target representation structures complete and empirically adequate.

Implicit versus explicit characterization of complex entities and events

A wide variety of relationships and behavioral dependencies can be seen abstractly as associative knowledge amongst individuals of the same complexity. Conversely, part-whole knowledge defines the relationships and behavioral dependencies between individuals of different complexity, arranged in part-whole hierarchies. Most of the current modeling paradigms do not correlate the two kinds of knowledge. It can be argued, rather, that an ontological dependency exists between associative and part-whole knowledge. Complex structured entities, referred to in the paper as wholes, are then required to explicitly encapsulate associative knowledge, thus providing units of reuse and stability. The principle is applied to behavioral modeling in Statecharts, where entity synchronization represents the associative knowledge used to compose complex behaviors from simpler ones.

Types of hierarchy imply types of model.

The types of hierarchy that describe the knowledge structures used by people doing complex tasks imply that certain types of cognitive mechanism are available to process them. This paper will discuss principles of organization in is-a (classification) and part-whole hierarchies. The principles of organization may be different at different levels of a hierarchy. The behaviour of people doing complex tasks is adaptable as a function of context. These factors taken together imply that cognitive processing is done by independent modules working within a context, rather than by a sequence of processing stages, and that a sequential stages processing model is a special case of a more general process. The paper ends with some practical implications of this change in models of the cognitive processes underlying complex behaviour.

Mapping part-whole hierarchies into connectionist networks

Three different ways of mapping part-whole hierarchies into connectionist networks are described. The simplest scheme uses a fixed mapping and is inadequate for most tasks because it fails to share units and connections between different pieces of the part-whole hierarchy. Two alternative schemes are described, each of which involves a different method of time-sharing connections and units. The scheme we finally arrive at suggests that neural networks have two quite different methods for performing inference. Simple “intuitive” inferences can be performed by a single settling of a network without changing the way in which the world is mapped into the network. More complex “rational” inferences involve a sequence of such settlings with mapping changes after each settling.

シーン概念における全体‐部分の階層構造

Two experiments were conducted to investigate the part-whole hierarchical organization of scene concepts. In each experiment, 16 subjects viewed a sentence “A is in B” and decided whether the scene A was a part of the scene B. The words used in Experiment 2 were less ambiguous than those used in Experiment 1. The ratio of correct responses and the reaction times from both of the experiments revealed that the part-whole hierarchy of scene concepts were different from concrete-abstract hierarchies of biological or object concepts in the following two respects. (1) The subjects' judgments did not necessarily follow a general principle of transitivity. That is, even if global-middle, and middle-local relationships were accepted, the relationship between global and local concepts was not always accepted, (2) The higher (more global) the level of concepts, the more fuzzy was the relationship between the concepts in hierarchy. These results were discussed in terms of the inherent concreteness of our knowledge about scenes.