Semi-lexical languages: a formal basis for using domain knowledge to resolve ambiguities in deep-learning based computer vision

Abstract

• Introducing semi-lexical languages for resolving ambiguities in Deep Learning-based classification using symbolic reasoning. • Ambiguities in Deep Learning-based classification of the parts are resolved by symbolic reasoning about the whole. • Demonstrating through suitable case studies that the semi-lexical framework surpasses or perform comparatively in terms of F1 score. • The semi-lexical structure enables explainable classification by providing both supportive and counter-factual arguments. Human vision is able to compensate imperfections in sensory inputs from the real world by reasoning based on prior knowledge about the world. Deep learning has had a significant impact on computer vision due to its inherent ability in handling imprecision , but the absence of a reasoning framework based on domain knowledge limits its ability to interpret complex scenarios. We propose semi-lexical languages as a formal basis for reasoning with imperfect tokens provided by the real world. The power of deep learning is used to map the imperfect tokens into the alphabet of the language, and symbolic reasoning is used to determine the membership of input in the language. Semi-lexical languages have bindings that prevent the variations in which a semi-lexical token is interpreted in different parts of the input, thereby leaning on deduction to enhance the quality of recognition of individual tokens. We present case studies that demonstrate the advantage of using such a framework over pure deep learning and pure symbolic methods.