CKTN: Commonsense knowledge transfer network for human activity understanding

Abstract

In most human activity understanding research, various deep learning networks are well designed to fit the character distribution of different human activities in images. After being fed with enough training data, the networks can perceive related patterns passively. Such abilities originate from the stimulation of training data, but these networks lack the ability to perform subjective judgment. When lacking sufficient training data, networks have difficulty perceiving valuable information, which leads to poor performance. However, human beings can infer human activities in complex situations through subjective judgment with their knowledge. On this point, we propose the commonsense knowledge transfer network (CKTN), which is empowered with human cognitive mechanisms and is able to understand human activities by using prior commonsense knowledge. More specifically, we study the structural combination rules of human bodies and explore the co-occurrence relationship between different body parts and objects. Then, we construct the part action cooccurrence probability matrix and human skeleton knowledge matrix, which work as prior knowledge, guiding our network to absorb and reason what the human activities in images are. Compared with traditional deep learning methods, our network realizes the unification of both subjective cognition and passive perception abilities. Experiments show the effectiveness of this method.