Spiking neural network based on joint entropy of optical flow features for human action recognition

Abstract

In the recent past, human action recognition is inviting increased attention in the automated video surveillance systems. An efficient human action classification technique in an unconstrained environment is proposed in this paper. A novel descriptor relying on joint entropy of difference in magnitude and orientation of the optical flow feature is developed in order to model human actions. Initially, flow feature is computed using Pyramid–Warping–Cost volume Network (PWCNet), considering every two consecutive frames. Then, the feature descriptor is formed based on the joint entropy of difference in flow magnitude and orientation collected from the regular grid of each frame in the action sequence. Finally, in order to incorporate long-term temporal dependency, a spiking neural network is embedded to aggregate the information across the frames. Different optimization techniques and different types of hidden nodes are utilized in the spiking neural network to analyze the performance of the proposed work. Extensive experiments on the benchmark datasets for human action recognition show the efficacy of the proposed method.