Humans can perform an enormous number of actions like running, walking, pushing, and punching, and can perform them in multiple ways. Hence recognizing a human action from a video is a challenging task. In a supervised learning environment, actions are first represented using robust features and then a classifier is trained for classification. The selection of a classifier does affect the performance of human action recognition. This work focuses on the comparison of two structures of the neural network, namely, feed forward neural network and cascade forward neural network, for human action recognition. Histogram of oriented gradients (HOG) and histogram of optical flow (HOF) are used as features for representing the actions. HOG represents the spatial features of the video while HOF gives motion features of the video. The performance of two neural network architectures is compared based on recognition accuracy. Well-known publically available datasets for action and interaction detection are used for testing. It is seen that, for human action recognition applications, feed forward neural network gives better results in terms of higher recognition accuracy than Cascade forward neural network.