Multi-input 1-dimensional deep belief network: action and activity recognition as case study

Abstract

This paper develops a new variation of deep belief networks which is evaluated on the basis of supervised classification of human actions and activities. The proposed multi-input 1-dimensional deep belief network (M1DBN) can work based on three inputs which contain different information structures. The multi input features helps M1DBN automatically search the solution space more accurately and extract high-level representations more efficiently. M1DBN utilizes three inputs to provide spatial, short-term and long-term information for the action and activity recognition. Spatial information can distinguish between human movements which have a high inter-class variation. However, regarding similar inter-class variations, a temporal description is used. Short-term and long-term inputs learn actions or activities for short and long video intervals, respectively. Experimental results show the superiority of this approach over state-of-the-art methods on KTH (97.04%), HMDB51 (67.19%), UCI-HAD (97.16%) and Skoda (93.28%) datasets. Also, a detailed explanation of learning, training and test procedures used in M1DBN are available.