3D Scene interpretation by combining probability theory and logic: The tower of knowledge

Abstract

We explore a newly proposed system architecture, called tower of knowledge (ToK), in the context of labelling components of building scenes. The ToK architecture allows the incorporation of statistical feature distributions and logic rules concerning the definition of a component, within a probabilistic framework. The maximum likelihood method of label assignment is modified by being multiplied with a function, called utility function, that expresses the information coming from the logic rules programmed to the system. The logic rules are designed to define an object/component by answering the questions “why” and “how”, referring to the actions in which a particular object may be observed to participate and the characteristics it should have in order to be able to participate in these actions. Two sets of measurements are assumed to be available: those made initially for all components routinely, and which supply the initial statistically based inference of possible labels of each component, and those that are made in order to confirm or deny a particular characteristic of the component that would allow it to participate in a specific action. A recursive version of the architecture is also proposed, in which the distributions of the former types of measurement may be learnt in the process, having no training data at all. Multi-view images are used as input to the system, which uses standard techniques to build the 3D models of the buildings. The system is tested on labelling the components of 10 3D models of buildings. The components are identified either manually, or fully automatically. The results are compared with those obtained by expandable Bayesian networks. The recursive version of ToK proves to be able to cope very well even without any training data, where it learns the characteristics of the various components by simply applying the pre-programmed logic rules that connect labels, actions and attributes.